Working with Unlabeled Data – Cluster Analysis¶
Find the best number of clusters with k_means and agglomerative clustering
Overview¶
- Load the data file
- check the shape and plot the content
- Observe the pair plot and comment the shapes in view of clustering
- if necessary, transform the data
- Use the elbow method to find the optimal number of clusters, to do this test
KMeanswith varying number of clusters, from 2 to 10: for each value ofk- fit the data
- compute the inertia and the silhouette score
- store them for plot
- Plot inertia and silhouette score versus
k - Choose the optimal number of clusters looking at the plots
- Cluster the data using the optimal number, plot the cluster assignment
- in the plot choose the features that seem to be most promising
- For comparison, repeat the same operation with the
AgglomerativeClusteringandDBSCAN
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score, silhouette_samples
from mpl_toolkits import mplot3d
from sklearn.model_selection import ParameterGrid
import warnings
warnings.filterwarnings("ignore")
random_state = 42 # This variable will be used in all the procedure calls allowing a random_state parameter
# in this way the running can be perfectly reproduced
# just change this value for a different experiment
1. Load the data file¶
Check the shape and plot the content
2. Observe the data distributions¶
We observe that the distributions of values are definitely skewed: in the columns from Fresh to Delicassen the values are highly concentrated on the right, but there are always outliers, frequently in a very large range.
Clustering is more effective in absence of outliers and with all the variables distributed in similar ranges, for this reason, we will execute two transformations:
- transform all the variables from the column
Freshto the columnDelicassenwith PowerTransformer - remap all the variables in the range
0:1
from sklearn.preprocessing import PowerTransformer
from sklearn.preprocessing import MinMaxScaler
Show the result of the transformation
Now the effect of outliers is reduced, and we compute the clustering
3. Use the elbow method to find the optimal number of clusters¶
Test KMeans with varying number of clusters, from 2 to 10
Prepare the results list that will contain pairs of inertia and silhouette_score for each value of k, then, for each value of k
- initialize an estimator for
KMeans - fit the data and predict the cluster assignment for each individual with
fitandpredict - the inertia is provided in the attribute
inertia_of the fitted model - compute the silhouette score using the function
silhouette_scorefromsklearn.metricsusing as arguments the data and the fitted labels, we will fill the variablesilhouette_scores - store the two values above in the list created at the beginning
4. Plot inertia and silhouette score versus k¶
5. Cluster with the optimal number¶
The two elbow points of inertia would suggest as cluster number 3 or 4, slightly more pronounced in 3. Silhouette has a maximum on 4, but the increase with respect to 3 is very small.
We will choose k=4
Show the distribution of samples in the clusters with a pie chart
Comments¶
The silhouette score ranges from -1 (worst) to 1 (best); as a rule of thumb, a value greater than 0.5 should be considered acceptable.
Agglomerative clustering¶
We will try a grid of parameter configurations, with the number of clusters in the range 2:10 and the four linkage methods available in the sklearn implementation of AgglomerativeClustering.
from sklearn.cluster import AgglomerativeClustering
The top five results have a very similar silhouette score, we will choose the setting with 4 clusters, as for k-means, and the linkage giving the best result with 4 clusters, that is ward. This is the result record with index 2 (the record index is the unnamed column at the very left of the dataframe output
Show the distribution of data in the clusters
Comments¶
The solution with the Agglomerative Clustering in this case provides a result very similar to that of kmeans.
It is interesting to compare more deeply the results of the two clustering models.
The function pair_confusion_matrix computes the number of pairs of objects that are in the same clusters or in different clusters in two different clustering schemes.
The result is given in a 2x2 matrix, the smaller the numbers out of the main diagona, the better the match.
For easier readability, we divide all the elements of the matrix by the sum of all the elements of the matrix, in this way, the matrix elements are normalized to 1.
from sklearn.metrics import pair_confusion_matrix
A short indicator of the match percentage can be optained as a sum of the elements of the main diagonal.
DBSCAN¶
from sklearn.cluster import DBSCAN
Show the distribution of data in the clusters
5. Find the best parameters using ParameterGrid¶
prepare the dictionary of the parameter ranges and produce the list of parameter settings to test with the function ParameterGrid
Arrange DBSCAN results in a dataframe, for easier presentation and filtering
dbscan_out = pd.DataFrame(columns = ['eps','min_samples','n_clusters','silhouette', 'unclust%'])
- for each parameter setting
6. Observe¶
- Observe visually the most promising combination of parameters reproducing the same number of clusters of the previous experiments
- fit and predict the cluster labels and show a pairplot
Show the cluster sizes
Silhouette plots¶
Use the function plot_silhouette contained in the module plot_silhouette_w_mean (provided with the notebook) and from sklearn.metrics import the function silhouette_samples providing the silhouette score for each sample
from plot_silhouette_w_mean import plot_silhouette # python script provided separately
Hint: use help(plot_silhouette) for the meaning of the parameters
Hint: for DBSCAN you should exclude the rows of noise
from plot_silhouette_w_mean import plot_silhouette # python script provided separately
Optional - Comparison with boxplots¶
For each of the clustering schemes show how the attribute values are distributed in the clusters
Control questions¶
- Repeat the experiments without the data transformations and comment the result
- Repeat the final fittings with the numbers of clusters immediately before and after the chosen values and comment the results